As the scaling of Large Language Models (LLMs) has dramatically enhanced their capabilities, there has been a growing focus on the alignment problem to ensure their responsible and ethical use. While existing alignment efforts predominantly concentrate on universal values such as the HHH principle, the aspect of culture, which is inherently pluralistic and diverse, has not received adequate attention. This work introduces a new benchmark, CDEval, aimed at evaluating the cultural dimensions of LLMs. CDEval is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains. Our comprehensive experiments provide intriguing insights into the culture of mainstream LLMs, highlighting both consistencies and variations across different dimensions and domains. The findings underscore the importance of integrating cultural considerations in LLM development, particularly for applications in diverse cultural settings. Through CDEval, we aim to broaden the horizon of LLM alignment research by including cultural dimensions, thus providing a more holistic framework for the future development and evaluation of LLMs. This benchmark serves as a valuable resource for cultural studies in LLMs, paving the way for more culturally aware and sensitive models.

This paper explores the use of clustering methods and machine learning algorithms, including Natural Language Processing (NLP), to identify and classify problems identified in credit risk models through textual information contained in validation reports. Using a unique dataset of 657 findings raised by validation teams in a large international banking group between January 2019 and December 2022. The findings are classified into nine validation dimensions and assigned a severity level by validators using their expert knowledge. The authors use embedding generation for the findings' titles and observations using four different pre-trained models, including "module\_url" from TensorFlow Hub and three models from the SentenceTransformer library, namely "all-mpnet-base-v2", "all-MiniLM-L6-v2", and "paraphrase-mpnet-base-v2". The paper uses and compares various clustering methods in grouping findings with similar characteristics, enabling the identification of common problems within each validation dimension and severity. The results of the study show that clustering is an effective approach for identifying and classifying credit risk model problems with accuracy higher than 60\%. The authors also employ machine learning algorithms, including logistic regression and XGBoost, to predict the validation dimension and its severity, achieving an accuracy of 80\% for XGBoost algorithm. Furthermore, the study identifies the top 10 words that predict a validation dimension and severity. Overall, this paper makes a contribution by demonstrating the usefulness of clustering and machine learning for analyzing textual information in validation reports, and providing insights into the types of problems encountered in the development and validation of credit risk models.

The documentation practice for machine-learned (ML) models often falls short of established practices for traditional software, which impedes model accountability and inadvertently abets inappropriate or misuse of models. Recently, model cards, a proposal for model documentation, have attracted notable attention, but their impact on the actual practice is unclear. In this work, we systematically study the model documentation in the field and investigate how to encourage more responsible and accountable documentation practice. Our analysis of publicly available model cards reveals a substantial gap between the proposal and the practice. We then design a tool named DocML aiming to (1) nudge the data scientists to comply with the model cards proposal during the model development, especially the sections related to ethics, and (2) assess and manage the documentation quality. A lab study reveals the benefit of our tool towards long-term documentation quality and accountability.

There is increasing evidence of the business benefits of responsible AI (RAI), when companies mitigate risks through training and testing data, measuring model bias and accuracy, and model documentation. Companies that adopt responsible AI experience higher returns on their AI investment. Raj Shekhar writes that business leaders globally must coalesce around the imperative to develop rigorous, consistent standards for responsible AI adoption. Much has been spoken and written about the risks to public trust and safety arising from the adoption of artificial intelligence (AI)-based applications across multiple sectors. In finance, the use of AI has led to discriminatory credit decisions.

If you came here looking for an introduction to water, or a synopsis of the 2003 TV series about teenage mermaids you have sadly come to the wrong place. The H2O that we will talk about is H2O.ai, a company which develops products for easy, scalable, machine learning and artificial intelligence. Machine learning and artificial intelligence (or AI for short) are topics which have had a lot of interest over the past 4-5 years. Some of this interest has come from businesses as they begin to utilise the information they collect on a day-to-day basis to streamline/automate processes or gain insight. A lot of companies are now looking to hire data scientists/engineers and in turn this is making a lot more people interested in machine learning and AI.

But how can organizations developing ML models enforce explainability and transparency standards when doing so might mean sharing with the public the very features, data sets, and model frameworks that represent that organization's proprietary intellectual property (IP)? Given machine learning's complexity and interdisciplinary nature, executives should employ a wide variety of approaches to manage the associated risks, which include building risk management into model development and applying holistic risk frameworks that leverage and adapt principles used in managing other types of enterprise risk. Whereas standard technical documentation is created to help practitioners implement a model, documentation focused on explainability and transparency informs consumers, regulators, and others about why and how a model or data set is being used. Such documentation includes a high-level overview of the model itself, including: its intended purpose, performance, and provenance; information about the training data set and training process; known issues or tradeoffs with the model; identified risk mitigation strategies; and any other information that can help contextualize the technology. Similarly, model documentation can become the proxy for sharing the model and its features and data sets with the world as opposed to sharing the actual "cookie recipe."